Add Ollama-webui package and service for Mixtral #275448

malteneuss · 2023-12-19T16:38:55Z

Description of changes

Related to #273556:

Add Ollama-WebUI package that mimics ChatGPT's frontend and integrates nicely with existing Ollama package.
Extend the Ollama service module for NixOS with options for home lab deployments
Add Ollama-WebUI service module for NixOS

Background:
To make open-source large langue models (LLM) accessible there are projects like Ollama that make it almost trivial to download and run them locally on a consumer computer.
We already have Ollama in Nixpkgs, but that can only be run conveniently in a terminal (and doesn't store previous chats). What's missing is a web UI, e.g. Ollama-WebUI that mimics ChatGPT's frontend and integrates nicely with Ollama.

Things done

Add a 👍 reaction to pull requests you find important.

nixos-discourse · 2023-12-19T21:13:28Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-package-static-single-page-nodejs-webapp-ollama-webui/37074/1

trzpiot · 2024-01-10T12:06:11Z

Hi @malteneuss, thanks for packaging Ollama Web UI. 💙 In the meantime, Ollama has been added as a service (but I haven't tested the service yet).¹ What's missing for bringing the Ollama Web UI to Nixpkgs? 😊

https://github.com/NixOS/nixpkgs/blob/nixos-unstable/nixos/modules/services/misc/ollama.nix ↩

mschwaig · 2024-01-12T10:53:15Z

As-is this does not build, but I have been running this PR with a few changes to get it to build locally for a few days and it's been working well. I have not looked at what would be required to land it in nixpkgs.
https://github.com/mschwaig/nixpkgs/tree/ollama-mixtral-services

To make open-source large langue models (LLM) accessible there are projects like Ollama that make it almost trivial to download and run them locally on a consumer computer. We already have Ollama in Nixpkgs, but that can only be run conveniently in a terminal (and doesn't store previous chats). What's missing is a web UI, e.g. Ollama-WebUI that mimics ChatGPT's frontend and integrates nicely with Ollama.

Make it convenient to setup Ollama-Webui on a server to run large language models (LLM).

malteneuss · 2024-01-12T21:32:51Z

@trzpiot I wanted to add an automatic NixOS test (since the ollama-webui needs to closely follow ollama; otherwise it broke a few times), but i haven't had time to learn to setup this up. Maybe a manual test would suffice for now to get it merged. Thanks for redirecting me to the existing ollama service (someone was faster ;) but i still would like to add some more knobs to make Ollama be deployable to a home lab server).

@mschwaig Thanks for testing it out. I pushed similar fixes to yours into the MR. I will try to test the setup during the weekend. Maybe you could do the same?

mschwaig · 2024-01-15T11:15:57Z

I am running this branch since yesterday.

Two things that I noticed were

that my existing model files disappeared. Probably this happens because I got ollama from the previous version of this branch before, and now I'm getting the one that got merged in the meantime and they store their models in different locations. I don't think that's an issue.
that long prompts made the connection between ollama-web-ui and ollama fail. I did not have enough time to look into this issue. If I have time I will test if the same thing also happens with oterm from this branch. I don't think thats an issue with this branch either.

Overall it seems to be running fine.

When you have added the extra config options that you still wanted to add I think you could remove the draft flag from this PR and at that point we can get someone to take a look who feels confident in reviewing the systemd services configs.

happysalada · 2024-01-18T14:06:29Z

nixos/modules/services/web-apps/ollama-webui.nix

+
+      ollama-webui-package = mkPackageOption pkgs "ollama-webui" { };
+
+      host = mkOption {


this host isn't used anywhere, is that intentional ?

happysalada · 2024-01-18T14:07:26Z

nixos/modules/services/web-apps/ollama-webui.nix

+      in {
+        ExecStart = "${cfg.ollama-webui-package}/bin/ollama-webui --port ${toString cfg.port} ${cors-arg}";
+        DynamicUser = "true";
+        Type = "simple";


just for information the default type is simple for services is simple. I'm not sure how I feel about having it written explicitely, if you prefer it that way, feel free to keep it.

happysalada · 2024-01-18T14:09:07Z

pkgs/tools/misc/ollama-webui/default.nix

+  npmDepsHash = "sha256-SI2dPn1SwbGwl8093VBtcDsA2eHSxr3UUC+ta68w2t8=";
+
+  # We have to bake in the default URL it will use for ollama webserver here,
+  # but it can be overriden in the UI later.


when you say it can be overridden later, does that mean inside the module ?
just wondering if you still need to add the override.

happysalada · 2024-01-18T14:13:32Z

nixos/modules/services/web-apps/ollama-webui.nix

+        Optional: This module is configured to run locally, but can be served from a (home) server,
+        ideally behind a secured reverse-proxy.
+        Look at <https://nixos.wiki/wiki/Nginx> or <https://nixos.wiki/wiki/Caddy>
+        on how to set up a reverse proxy.


do you think referencing the caddy file of the original repo would be a good idea ?
https://github.com/ollama-webui/ollama-webui/blob/main/Caddyfile.localhost

happysalada · 2024-01-18T14:15:07Z

pkgs/tools/misc/ollama-webui/default.nix

+  PUBLIC_API_BASE_URL = "http://localhost:11434/api";
+
+  # The path '/ollama/api' will be redirected to the specified backend URL
+  OLLAMA_API_BASE_URL = PUBLIC_API_BASE_URL;


the env var only seems to be set after the build has succeeded
https://github.com/ollama-webui/ollama-webui/blob/main/Dockerfile#L22
does the build fail without ?
if not, this is probably something that should be set in the service, not in the package.

happysalada · 2024-01-18T14:19:58Z

pkgs/tools/misc/ollama-webui/default.nix

+    mkdir -p $out/lib
+    cp -R ./build/. $out/lib
+
+    mkdir -p $out/bin


you don't need those last instructions.
I could be wrong, but if this is a standard node project, you just need to package the build directory
then in the service you can use node directly to just run the build directory
here are some reference of how we did this with lemmy

nixpkgs/pkgs/servers/web-apps/lemmy/ui.nix

Line 35 in aa5d1d0

name = "lemmy-ui";

nixpkgs/nixos/modules/services/web-apps/lemmy.nix

Line 4 in aa5d1d0

cfg = config.services.lemmy;

second point and this is entirely optional.
rather than making a complety separate service for ollama-webui, how about including it in the original ollama service.
I don't think it makes sense to have ollama-webui as a standalone for now.
(you can take inspiration in the lemmy module for what we did if you find stuff that you like).

just a bit more context on not providing an executable for a node package.
the main reason is that without several environment variable set, the binary is just not runnable. So most of the time you package the build directory and in the service you provide all the necessary to run it.

if you prefer to provide a binary, I respect your decision, in that case, you should probably use makeWrapper, you can look at jellyseerr for an example (you'll find many more if you don't like this particular one).

second point and this is entirely optional.
rather than making a complety separate service for ollama-webui, how about including it in the original ollama service.
I don't think it makes sense to have ollama-webui as a standalone for now.
(you can take inspiration in the lemmy module for what we did if you find stuff that you like).

Since ollama-webui is a community project and not officially associated with ollama (see the top of their README.md), I tend to think it does not make sense to have the ollama-webui service live under ollama as ollama.ui or ollama.web-ui.

There are a bunch of other frontends that people might want to use with the ollama service, such as oterm.

happysalada · 2024-01-18T14:24:36Z

hey, I've provided a couple of comments, but all in all, this is very nice! Thank you for your contribution. I'm here to help, so let me know if anything!

wochap · 2024-01-22T20:22:28Z

Hi, I just used your branch, and I noticed two things:

The services.ollama-webui.port doesn't work as expected. This is because the http-server within your script wrapper does not receive the arguments (--port). To fix this, add "$@" at the end of http-server within the script wrapper. For e.g.: https://github.com/wochap/nix-config/blob/dev/packages/ollama-webui.nix#L33
By default, http-server does not handle Single Page Applications well. If you try to access http://localhost:8080/c/<chat_id> and refresh, it will result in an error. A workaround is to add --proxy http://${host}:${port}? to http-server, more info https://github.com/http-party/http-server?tab=readme-ov-file#catch-all-redirect. However, it would be ideal to use vite --preview --outDir <path_to_ollama_webui_dist_folder>.

Thanks for creating this PR!

mschwaig · 2024-03-04T02:01:52Z

@malteneuss are you still interested in/do you still have time to move this forward? Would it be OK for someone else to pick it up?

For those who are interested in this functionality, I have a branch with a rough update of this branch to a more recent version of what's now called open-webui: https://github.com/mschwaig/nixpkgs/tree/open-webui
That branch is still in rough shape. I have no immediate plans to work on a PR (though that might change). Feel free to use it in this or another PR.

To me it looks like those more recent versions require also packaging some python (i think mostly for RAG), which blows up the scope a bit. Especially since some dependencies have not landed in nixpkgs yet.
I am also a bit concerned about some config options for privacy shipped as part of the Dockerfile, which is the kind of thing we would have to really pay attention to when updating this service: https://github.com/open-webui/open-webui/blob/eb51ad14e4caafda1c9fd24c4945044b8776a7a3/Dockerfile#L30C1-L31C22

malteneuss · 2024-04-06T20:51:49Z

@happysalada @mschwaig Thanks for the review comments and pushing this topic further. I've been busy with my full-time job and a two year old, and have little to no time to move this forward in the next months. Would your branch be stable and usable enough to be merged and improved in smaller steps?

Kreyren · 2024-05-07T21:30:22Z

nixos/modules/services/misc/ollama.nix

+        HOME = "%S/ollama";
+        OLLAMA_MODELS = "%S/ollama/models";


Isn't it better to use /var/lib/private/ollama instead of the %s so that it's more functional to work with the variables?

^ Having to hard-code the path as the expansion doesn't seem to work like other services

Kreyren · 2024-05-07T21:47:43Z

@happysalada @mschwaig Thanks for the review comments and pushing this topic further. I've been busy with my full-time job and a two year old, and have little to no time to move this forward in the next months. Would your branch be stable and usable enough to be merged and improved in smaller steps? -- @malteneuss (#275448 (comment))

Is this still on-going?

yes -> Please share the current expected status so that i know where to start from as i might have time to finish this MR
no -> cool cool

mschwaig · 2024-05-07T22:10:32Z

@Kreyren

As far as I remember the chat functionality is working on my branch (linked above).
I have not tested that RAG is working yet, which is the feature that made adding all that python necessary.
Some of the python dependencies had to be packaged, some had open PRs which might have been closed in the mean time.

I do not have a lot of time to look into this in the next two weeks.
My next steps would have been to take another look at my branch with the goal of opening a draft PR once I get to it, maybe rebase onto recent master and try to build a newer version of the package, but if someone else wants to pick this up and open a PR in the mean time, go ahead.

I would recommend that you use my branch as a starting point.

malteneuss · 2024-06-08T13:55:24Z

Already done with #316248

NixOSInfra added the 12. first-time contribution This PR is the author's first one; please be gentle! label Dec 19, 2023

eclairevoyant added the 2.status: work-in-progress This PR isn't done label Dec 19, 2023

ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Dec 19, 2023

malteneuss mentioned this pull request Dec 20, 2023

feat: create releases with ollama-webui version numbers open-webui/open-webui#255

Closed

malteneuss force-pushed the ollama-mixtral-services branch 8 times, most recently from a4d4021 to 13003f5 Compare December 22, 2023 21:00

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Dec 22, 2023

malteneuss force-pushed the ollama-mixtral-services branch 4 times, most recently from 9bb3767 to e07509d Compare December 22, 2023 22:57

ofborg bot removed the 10.rebuild-linux: 1 label Dec 23, 2023

nixos/ollama-webui: init service

d48979f

Make it convenient to setup Ollama-Webui on a server to run large language models (LLM).

malteneuss force-pushed the ollama-mixtral-services branch from e07509d to d48979f Compare January 12, 2024 21:25

happysalada reviewed Jan 18, 2024

View reviewed changes

h7x4 added the 8.has: module (new) This PR adds a module in `nixos/` label Jan 18, 2024

mschwaig mentioned this pull request Mar 19, 2024

Package request: ollama and ollama-webui as NixOS modules #273556

Closed

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 20, 2024

Kreyren reviewed May 7, 2024

View reviewed changes

malteneuss closed this Jun 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Ollama-webui package and service for Mixtral #275448

Add Ollama-webui package and service for Mixtral #275448

malteneuss commented Dec 19, 2023 •

edited

Loading

nixos-discourse commented Dec 19, 2023

trzpiot commented Jan 10, 2024

mschwaig commented Jan 12, 2024

malteneuss commented Jan 12, 2024

mschwaig commented Jan 15, 2024

happysalada Jan 18, 2024

happysalada Jan 18, 2024

happysalada Jan 18, 2024

happysalada Jan 18, 2024

happysalada Jan 18, 2024

happysalada Jan 18, 2024

happysalada Jan 18, 2024

mschwaig Jan 23, 2024

happysalada commented Jan 18, 2024

wochap commented Jan 22, 2024

mschwaig commented Mar 4, 2024

malteneuss commented Apr 6, 2024

Kreyren May 7, 2024 •

edited

Loading

Kreyren commented May 7, 2024

mschwaig commented May 7, 2024

malteneuss commented Jun 8, 2024


		ollama-webui-package = mkPackageOption pkgs "ollama-webui" { };

		host = mkOption {

Add Ollama-webui package and service for Mixtral #275448

Add Ollama-webui package and service for Mixtral #275448

Conversation

malteneuss commented Dec 19, 2023 • edited Loading

Description of changes

Things done

nixos-discourse commented Dec 19, 2023

trzpiot commented Jan 10, 2024

Footnotes

mschwaig commented Jan 12, 2024

malteneuss commented Jan 12, 2024

mschwaig commented Jan 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

happysalada commented Jan 18, 2024

wochap commented Jan 22, 2024

mschwaig commented Mar 4, 2024

malteneuss commented Apr 6, 2024

Kreyren May 7, 2024 • edited Loading

Choose a reason for hiding this comment

Kreyren commented May 7, 2024

mschwaig commented May 7, 2024

malteneuss commented Jun 8, 2024

malteneuss commented Dec 19, 2023 •

edited

Loading

Kreyren May 7, 2024 •

edited

Loading